Measuring Diversity in Regression Ensembles

The problem of combining predictors to increase accuracy (often called ensemble learning) has been studied broadly in the machine learning community for both classification and regression tasks. The design of an ensemble is based on the individual accuracy of the predictors and also how different they are from one another. There is a significant body of literature on how to design and measure diversity in classification ensembles. Most of these metrics are not directly applicable to regression ensembles since the regression task inherently deals with continuous valued labels for learning. To measure diversity in regression ensembles, Krogh and Vedelsby show that the quadratic error of an ensemble estimator is guaranteed to be less than or equal to average quadratic error of components. However, this does not give a way to measure or create diverse regression ensembles. This paper presents metrics (correlation coefficient, covariance, dissimilarity measure, chi-square and mutual information) that can be used for measuring diversity in regression ensembles. Careful selection of diverse models can be used to reduce the overall ensemble size without substantial loss in performance. We present extensive empirical results to show the performance of diverse regression ensembles formed by Bagging and Random Forest techniques.